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A procedure is presented for evaluating the performance of a general 
class of digital detectors which square or multiply signal waves contaminated 
by gaussian noise. In addition to simplifying and unifying the treatment 
of a number of previously solved problems and some hitherto unsolved ones, 
the method achieves a considerable advince toward a complete evaluation 
of postdetection filtering. In contrast to most of the earlier related work, 
which is typically restricted to filters described as accepting low-frequency 
difference products and rejecting high-frequency sum products, the present 
analysis offers a tractable inclusion of filters which do significant selective 
processing of the detected low-frequency signal and noise components. 

A principal goal sought is the asymptotic form approached by the error 
probability expressed as a function of the signal-to-noise ratio when the 
latter is large. This is a primary region of interest in digital data trans- 
mission over the telephone network, and the applicable results give a basis 
for comparing performance of different systems. The mathematical problem 
is one of calculating the probability that a quadratic form in a set of gauss- 
ian variables with arbitrary means and variances will assume values 
critically far removed from that obtained when each variable is at its mean. 
The mean values represent signal contributions unperturbed by noise and 
for good performance should dominate over the noise except at the tails of 
the distribution. Concentration of attention on the infrequent large noise 
peaks calls for an approach inherently different from the conventional 
series expansions appropriate near the center of the distribution. The 
results are of importance not only in detection theory bid also in general 
statistical analysis of rare events. 

I. INTRODUCTION 

A general class of data receivers have decision logic based on observing 
at the output the sign of a quadratic form 
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q = 5Z QijW.Wj ss w + Qw. 



(1 



Examples of such quadratic detectors and the reduction of their output 
to the form (1) are given in Sections IV and V, and a schematic repre- 
sentation in Fig. 1. Let it suffice for the present to say that the real 
symmetric matrix Q is determined by the system niters, while the real 
JV-dimensional vector w and its transpose w + are related to the received 
signal plus noise. 

During a considerable portion of this paper we shall be concerned 
with the asymptotic evaluation of the probability of error for such 
receivers when the noise is additive gaussian. To develop the tools 




Fig. 1 — Model of the quadratic detector. 

for this end, and also partly for general mathematical interest, Sections 
II and III are devoted to the following general problem: Given that 
the real gaussian vector w whose components have means w, and a 
positive definite* covariance matrix M, 



Mij = ((wi — Wi) (wj — ibj)) = Mji, 



(2) 



is such that the quadratic form w + Qw is positive, what is the probability 
P e that for small noise (i.e., that the w, have small variances) the quad- 
ratic form (1 ) is negative? We refer to P e as the asymptotic probability 
of error, f Various special cases of this problem have been treated in 
the literature; some references may be found in Section IV where we 
reproduce and generalize some of these special results. 

* Without any real loss of generality strict positive definiteness is assumed, 
f As pointed out later our techniques may also be used to obtain the distribu- 
tion function in certain regions, not only the probability of error. 
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A systematic treatment of the general problem has hitherto been 
lacking. Progress has been inhibited due to the fact that the exact 
probability distribution of arbitrary gaussian quadratic forms is too 
complicated to be useful. Experience has shown that the curves for the 
probability of error vs signal-to-noise ratio tend to be roughly parallel 
for different data receivers and to be characterized sufficiently well by 
their asymptotic slopes. We, therefore, do not strive to obtain exact 
error rates but rather deal with asymptotic forms for large signal-to- 
noise ratios. In other words, we will be concerned with the behavior of 
the distribution on the tails of the density functions of our quadratic 
forms. For high error rates or low signal-to-noise ratios, other approxi- 
mations may be obtained by using various well-known moment series 
such as the Edgeworth and Gram-Charlier expansions. 

We show in Section V how our results may be used to attack the 
problem of the distribution of the filtered response of a product detector. 
It is primarily this consideration of postdetection filtering that has been 
absent from earlier discussions. Finally in Section VI, a simple model 
of a fairly representative class of quadratic detectors is analyzed in 
some detail by means of a rapidly convergent expansion in prolate 
spheroidal wave functions. 

II. GENERAL ANALYSIS FOR QUADRATIC FORM 

In the introduction we defined the problem of obtaining the prob- 
ability of error for the quadratic form q. We approach this problem via 
the characteristic function C(u) of (1), which is well known to be given 
by' 

cM -< e >•- vdet (/ - mm ' 

The symbol (■)« denotes the average with respect to q; "det" means 
determinant and / is the identity matrix. Since ilf -1 is the inverse of a 
real symmetric and positive definite matrix, it itself has all of these 
properties. We now note the following theorem: 2 Let A and B be real 
symmetric matrices, and further let A be positive definite. Then there 
exists a real matrix S such that 

S + AS = / (4) 

S + BS = a 2 D, (5) 

where D is some diagonal matrix and a 1 is a positive parameter intro- 
duced for later convenience. 
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Equation (4) implies, in particular, that 5 -1 exists. If we identify A 
with M~ l , and B with Q, (4) and (5) tell us that we may write 

M = SS + (4a) 

Q = (S + rVDS~\ (5a) 

If we substitute (4a) and (5a) into (3), and further introduce a new 
real gaussian vector y by the linear transformation 

y = S-'w, (6) 

we arrive at a simpler expression for the characteristic function, namely, 

Qf \ = exp [\y + (I - 2iua 2 D)~ l y - \y + y\ , . 

Vdet (/ - 2ia><r 2 D) 

The principal simplification is now that D is a diagonal matrix. We note 
that 

w + Qw = <r 2 y + Dy (8) 

and 

w + Qw = ay + Dy. (9) 

We find it more convenient to deal with q when it is expressed in terms 
of the variables y t . 

The probability of error denned earlier may be expressed in terms of 
the characteristic function by the formula 

Pr{g<0} -P e = -l-.T-^ldco. (10) 

2iri J-* co + *« 

The ie (e > 0) appearing in the denominator of the integrand and in 
(10) is used to signify that in the complex co-plane, the contour of in- 
tegration implied in (10) goes above the singularity at u = 0. Making 
use of (7), we write (10) in more detail* 



j, * > d<* ' 



= --( 

2wi J- 



2iri J- oo co + it 



''' y ^(l-2^W 2 dy) 



(id 



* Henceforth, we will not use bars to denote the mean of a random variable. 
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2iri J- 



dw 



2iri X-oo to + *'« 



exp 



~2?pi i 

CO + - 



n (i - wdt) 

2/ 2 



(12) 



where dj is thej'th element of the real diagonal matrix D. Equation (12) 
follows from (11) by a simple change of variable. We observe in (12) 
that the quantity l/o- 2 appears only in the exponent, and therefore we 
may obtain an asymptotic expansion of P e valid for small a by con- 
sidering only the exponent. One will see in later sections that this 
asymptotic result for small a corresponds to the asymptotic result for 
large signal-to-noise ratios. 

Proceeding with the analysis of (12), we remark that the integrand 
obviously falls off sufficiently rapidly at infinity to allow one to close or 
distort the contour at infinity without changing the value of the in- 
tegral. We further note the singularities of the integrand of (12). There 
is a simple pole at co = which has already been discussed. In addition 
to the simple pole at to = 0, the exponent has simple poles at to = — i/d t ; 
these all lie along the imaginary axis. Also at these points the denomina- 
tor of the integrand has, in general, branch points due to the square 
root. For doubly degenerate eigenvalues, the branch points become 
simple poles. 

We now concentrate our attention on performing the integration in 
(12) for small a by the saddle point method. ' To locate the saddle 
points, let to = in, and consider the solutions to 



d_ 



E 



Vi 



+i 



= o, 



13) 



or 



F(n) - - E 



d; 



'(•♦sy 



= 0. 



(14) 



Consider F( n ) defined by (14) and let n be real. We shall show that 
there always exists a saddle point for positive a (to positive imaginary), 
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which occurs before the first singularity of the exponent on the positive 
imaginary axis.* We separate the sum (14) into two parts 

2 2 

We note that when « « — (l/d»), F(n) is large and positive for d, < 
and F(n) is large and negative for dt > 0. Also note that 
F(0) = —^2d;yi 2 ^ by (9) and our assumption that the noise-free 
signal is positive. Hence by continuity, there must be at least one n > 
for which F( n ) = 0, which locates a saddle point for us. Clearly by 
the monotonicity of the two parts of expression (15) there can be only 
one such saddle point between the origin and the first singularity of the 
exponent on the positive imaginary axis. 

The relations in the complex co-plane described above are illustrated 
in Fig. 2. The dotted line labeled Li is the original contour of integration, 
and the small circle labeled "S.P." is the (unique) saddle point on the 
positive imaginary axis lying before the nearest singularity w = — i/d, , 
di < 0, which has nonvanishing residue ?/,- in the exponent of the inte- 
grand. Also drawn on the imaginary axis in Fig. 2 are the branch cuts 
of the denominator, indicated by the hatched bars in the figure. We 
assume in this section (and Fig. 2 is so drawn) that the saddle point 
occurs before the nearest singularity a> = — i/d on the positive imagi- 
nary axis.f As discussed above, this is guaranteed to be the case if only 
(y') 2 = 0.t 

If the saddle point is situated as shown in Fig. 2, the contour may 
then be shifted from the real axis (line L\ in Fig. 2) to a contour which 
is a straight line parallel to the real axis and passing through the saddle 
point (the line Li in Fig. 2). The integrand drops off sufficiently rapidly 
for large | « | so that the ends of the contour connecting L x and L 2 (in 
accordance with Cauchy's theorem) give no contribution. We will now, 
for large signal-to-noise ratios (small <r 2 ), approximate the integral 
along the contour L 2 by the contribution in the immediate neighbor- 
hood of the saddle point. It is shown in the appendix that the magnitude 

* This assumes that at least one of the di is negative with nonvanishing residue 
yi 1 . We also, of course, are assuming that the quadratic form q in the absence of 
noise is positive. 

t For definiteness we have based our analysis on the assumption that the noise- 
less signal is positive. We emphasize that the role of upper and lower half planes 
would be interchanged if one assumed that the noiseless signal were negative. 

\ We have used d' to denote that particular di which corresponds to the nearest 
singularity on the positive imaginary axis; y' is the associated y,- . 
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S.P. 



Fig. 2 — Contours and singularities in the co-plane. 

of the exponential term in the integrand of (12) is a monotonically 
decreasing function of w as one recedes from the saddle point on the 
contour L 2 , and therefore this saddle point evaluation is asymptotically 
correct. 3,4 One might also add that it can be shown that the contour L 2 
is in fact tangent to the path of steepest descent at the saddle point. 

To obtain an explicit formula for our asymptotic evaluation of P c 
under these conditions, write (12) as 



P e = 



1 f ( \ r/^~L, 

- - o— • / 9W ex P ~ ~T dz ' 



with 



and 



9(z) «vH(i-**) 



''-IS* 

a- 



(16) 



(17) 



(18) 
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Then, by a standard saddle point evaluation, we have 
_1 1 <r 

Pe ~ V** rVIIO + rd,-) VFW) exp 






(19) 



where, to repeat, T is the smallest positive root of the equation 

£/(*)U*r = 0. (20) 

dz 

The notation /*(*T) in (19) means the second derivative oif(z) evalu- 
ated at z = iT. 

We finally wish to note that although we have concentrated in this 
section on the evaluation of P e , one could use entirely analogous tech- 
niques to find an asymptotic expression for the probability P K that the 
quadratic form g is less than some number K as long as K is less than 
the value of q in the absence of noise. The analog of (10) is (for any 
value of K) 

P K = - k—. \ — T-^e da). (21) 

ZTTl J-oo oj -\- It 

For a K satisfying the stated condition, the integrand in (21 ) possesses 
a saddle point such that a discussion for asymptotically calculating P K 
may be given which is entirely analogous to that already given for P e . 
For a given a- 2 , the accuracy of such an approach will depend on K. 

III. VANISHING OF CRITICAL RESIDUE IN UPPER HALF-PLANE 

We wish, in this section, to review a particularly trivial examine to 
illustrate a violation of a condition necessary for the guaranteed ap- 
plicability of our method, namely the nonvanishing of the residue as- 
sociated with the nearest pole in the upper half-plane. In the example 
we consider here, the exact answer can be obtained by a simple contour 
integral. Consider the quadratic form 

z x = Xi — xi + xz — x 2 , (22) 

where the z's are independent gaussian variables, all with the same 
variance a. Further, assume that x 2 and x± have zero means. Then the 
characteristic function of z is 

. Xi 2 + x 3 2 
exp to} _ ~ . 2 

C{o3) = (l-2iW^)(l+ C 2W)' (23) 
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The contour integral implied in (10) may be closed in the upper half- 
plane to yield immediately by exact methods 



P e = fexp - 



r-**4 



(24) 



The exponent in (24) is simply obtained by evaluating the exponent in 
(23) at the singularity oj = i/2<r 2 in the upper half-plane. Now consider 
instead the expression 



22 = 2l + 2 £2n+l Z — 2 ^fci , 



(25) 



where z 2 in the absence of noise is assumed positive, the additional 
gaussian variables are assumed to have variances which are less than 
a 2 , and the variable .r 2 „ (n ^ 3 ) with smallest variance has non vanishing 
mean. The exponent [— f(iy)/<r*\ for the characteristic function of 2 2 
as one travels up the positive imaginary axis appears as in Fig. 3, where 
we have written w = iy. Two situations are clearly possible. First y < 
l/2tr 2 ; in this case the contour of integration for (10) may be distorted 
to pass through the saddle point, and the previous discussion then ap- 
plies. Since .r 2 2 + x* = this could not be guaranteed a priori. The 
second case is y > l/2<r 2 ; in order to distort the contour to pass through 
the saddle point in this situation, one must first sweep the contour past 
the singularity at a> = i/2a 2 . Since this singularity is, for the present 
case, only a simple pole, the result of pushing the contour past is simply 
to pick up the residue of this pole. As an asymptotic answer one then 
has this residue plus the saddle point contribution. However, it is clear 
from Fig. 3 that the residue term (recall yo > l/2<r 2 ) has a less negative 
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Fig. 3 — Behavior of exponent discussed in section III. 
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exponent, and thus in the limit of large signal-to-noise ratio will be ex- 
ponentially dominant over the saddle point. If, for this second case, one 
has instead one or more branch points before the saddle point, the 
analysis is not as simple, but realizing that one may distort up to the 
first singularity (which, of course, has vanishing residue in the expo- 
nent), and keeping in mind Fig. 3, one might argue that the exponential 
behavior for this case is determined by the value of the exponent evalu- 
ated at the nearest singularity in the upper half-plane. 

IV. SIMPLE COMMUNICATION APPLICATION 

In this section we apply our saddle point technique to a number of 
problems whose asymptotic forms have appeared previously in the 
literature with derivations based on techniques different from ours. 
The reproduction of previous results helps to establish confidence in 
our methods, and arrives at these answers in a more straightforward 
manner than previously. 

The problems considered here may all be put into the form where 

2 i 2 2 2 2 2 /nn \ 

q = U\ + ih — Vi — v 2 = u — v . (26) 

That is, q is the difference of the squared lengths of two two-dimen- 
sional gaussian vectors. We let the variance of both components of the 
vector u be equal to a£ and of both those of v equal to <t\. Further, all 
components are independent. We shall see later that analysis of binary or 
multilevel FM using discrimination detection and differential detection 
of FM 9 reduces to this case with, in general, <n ^ <r 2 , while the analysis 
for differential phase modulation is also of this form, with a x = o- 2 . 
The characteristic function for q defined by (26) is simply 

. r__^ v 2 I 

nl v _ exp *" h - 2W 1 + 2zW 1 2 J (9 -s 

t{u) ~ (1 -2*W 2 ')(1 + 2W) ' K } 



and the probability of error is 



2 

oj V 



2m J- 



exp ~ W 



da 

2iri J- oo co + it 



+ 



- iA A 



(~a 



(28) 



where 



A = <72 Ai . (29) 
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Letting, again, « = iy, we find that the saddle point of interest is at 

u 2 + AV - uv{\ + A) 2 , , 

2/o = A - j^-r . {60) 

Using (19), we then obtain for the probability of error* 

1 o-! 1 (w - A 2 v)(u + AH) 



P, 



V2t \/l + A 2 \/uv [u + AV - uv(l + A 2 )] 
• eXP L2. 1 2 (l + A 2 )_- 



(31) 



We might note in passing that if one has two independent Af-dimensional 
gaussian vectors u N and v N and if all the components of u N (v N ) are 
independent and have the same variance ai (<ji ), then the probability 
of error for the difference of the square of these vectors, u N 2 — v N 2 , will 
have different multiplicative factors from these in (31) but the expo- 
nent in (31) will still be correct when u and v are reinterpreted to be the 
lengths of the vectors u^ and v N , respectively. The result (31) and its 
generalizations for higher dimensions may also be viewed as giving the 
asymptotic form of the cumulative probability distribution for the 
doubly noncentral F-distribution. Exact, but not very transparent, 
formulas for this problem have recently been published by Price. 

Analysis of the error performance of binary FM and differential phase 
modulation leads one to consider the probability that the inner product 
q of two independent 2-dimensional gaussian vectors, a and 5, 

q = 2a- ? (32) 

is negative when the inner product of the means is positive. ' ' Here 
again the components of each vector are independent, and those of a 
have the same variance and those of (3 have possibly a different variance. 
However, it is clear that multiplication of (32) by a positive constant 
can adjust these two variances to be identical (with appropriate ad- 
justment of the means) without affecting the probability of error. 
Hence, it suffices to choose the variances to be equal to the same con- 
stant a 2 . Further, introduce 

U = (a + ?)/ \/2 

(33) 
v = (a- &)/V2, 

so that 

* If we write (31) as P c = f(u,v,ai,<ro), then, if the noisefree form (26) is nega- 
tive, the probability P c ' that q is positive is P e ' = —f{u,v,a\,ai). 
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q = u 2 - v\ (34) 

The conditions assumed in the derivation of (31) are satisfied by (34), 
and in particular we have 

o"i 2 — <r% = tr . (35) 

Equation (31) for the probability of error then simplifies to 

P e ~ — 7=— 1= exp — - — . (36) 

2Vir Vuv u — v e L 4<r 2 J 

By making extensive use of (33), it is readily verified that our expres- 
sion (36) for the asymptotic (large S/N) probability of error for q 
defined by (32) is identical with the expression given in Bennett and 
Davey, 6 (9-56). The considerably more complicated form of the Bennett- 
Davey result is solely due to the fact that they express their answer in 
terms of variables o and (3 instead of u and v. 

We now consider another application of our formula (31), namely to 
the analysis of errors in multilevel FM data transmission using dis- 
crimination detection. In this application, one is concerned with the 
probability that the instantaneous frequency \p is in error, where 4/ is 
given in terms of in-phase and quadrature components, x(t) and y(t) 
respectively, by the equation ' 

* = %^- (37) 

x 2 + y 2 

The quantities x, y, x, y are gaussian variables with arbitrary means, 
and with variances equal to a, a, & 2 , & 2 respectively. We also use the 
notation 

(38) 
RR = xx + yy. 

Suppose for the noise-free situation ^ > z. It is of interest in the multi- 
level situation to know the probability P that \j/ < z in the presence of 
gaussian noise. To put this problem into a form to which (31) is directly 
applicable, we made use of the following chain of equalities : 

P = Pi# ^ z\ = Pi# - z ^ 0] 

= Pr[x(y - zx) + y(-x - zy) ^ 0] 
= Pr[xa + yb ^ 0], 



where 



a = k(y — zx) 

(39) 
b = -k(x + zy), 



and k is any positive constant. 
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(40) 



(41) 



(42) 

Then we see that 

P = Pr (u* + ui - v{ - v. 2 = 0), (43) 

where the set of variables (iii , u 2 , Vi , vo) are all independent (due to 
this choice of k) and further, 

<r Ui 2 = CT „; 2 = a 2 (l - l, z f + fcV = ai (44a) 

^ = av ,; 2 = a 2 (l + A:z) 2 + fcV = «7i 2 . (44b) 

It is useful to write u = u 2 + m 2 2 and v = V\ + *V in terms of the 
original FM variables. Using (37) — (40), we find that 

u' = R* {[l + kz (i - l)J + k" (§)] (45a) 

„' = tf{[l -*.(*- l)J + *'(|)} (45b) 

A convenient simplification results if we restrict ourselves to constant 
amplitude FM waves, i.e., R = 0. For this specialization we have 

u = R[l + fc(l£ - z)] (46a) 

» = 4W Z > "jf"1<! (46b) 

[«(^ — 2) — 1 if /o(i/' — 2) > 1. 

If we set 2 = 0, we immediately have that k = a/&, a 2 =a£ = 2a 2 , and, 
distinguishing the two cases in (46b), (46) and (31) reproduce exactly 
(38) and (39) of Ref . (6). 



2178 THE BELL SYSTEM TECHNICAL JOURNAL, NOVEMBER 1965 

Equations (46) and (31) together provide general formulas for the 
asymptotic evaluation of multilevel FM. 

V. GENERAL QUADRATIC DETECTOR 

In this section we consider the probability of error for the filtered 
response of the product detector given in Fig. 1 when the noise input is 
white gaussian. The signal s(l) plus the added noise n(l) is divided into 
two branches, each of which has a filter (denoted by F\ and F<> for the 
two branches). The outputs of these filters are multiplied and the 
product is passed through the filter F% , whose output is the final system 
output. This product detector is a generalization of the square-law 
detector considered by Kac and Siegert, 1 " and by Emerson. 1 ' 1 We first 
parallel Emerson's treatment and express the problem as in expression 
(1), except now N = °° . 

One familiar with Refs. 12 and 13 will not be surprised that the results 
amount to solving an integral equation, one which there is little hope 
of solving in practice. Therefore, we feel that an important point is 
made when we show in the next section that for a particular model of 
considerable practical interest, one can effectively approximate the 
system function by a kernel of finite rank, expressible in terms of known 
functions; the functions we have in mind for this purpose are the pro- 
late spheroidal wave functions. 1 We wish to emphasize that although 
the results of earlier sections are applied here, we regard the rapidly 
converging approximations to the system function as important in the 
treatment of this problem. 

Following Emerson, let /,(/) be the impulse response of the zth filter. 
Then the output at time t, E out (t), may be written in terms of the input 
wave 

E iD (t) = s(t) + n(t) (47) 

as 

EoM = / f du do E ia (t - u)g(u,r)E in (t - v) , (48) 
where the system function g(u,v) is given by 

g(u,v) = f fiiu - z)Mz)f 2 (v - z)dz. (49) 

•'—00 

One can immediately see from (49) that if the filters Fi and F 2 are 
not identical, then g(u,v) is not a symmetric function of u and v. How- 
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ever, it is apparent from (48) that if we write the identity 

g(u,v) = \[g{u,v) + g (»,«)] + \ [g(u,v) - g(v,u)], (50) 

then when (50) is used in (48), the second term in (50) gives no con- 
tribution, and we have 

E out (t) = jf du do E-Jt - u)G(u,v)E in (t - v), (51) 

where 

G(u,v) = h\g(u,v) +g(v,u)]. (52) 

The kernel G(n,v) is now hermitian, and all its eigenvalues are guaran- 
teed to be real; we also assume that G is square integrable. If X n denotes 
its nth eigenvalue and <p n (t) its nth eigenf unction, then we may write 
the well-known operator result* 

G(u,v) = '2\ n fn(u)ip n {v). (53) 

Thus, 

E out (t) = 2\„e n 2 (t), (54) 

where 

e„(t) = [ <Pn(v)E in (t - v) dv. (55) 

J— 00 

Upon using (47) and the fact that n(t) represents white gaussian noise 
with correlation function 

(n(t)n(t')) = N 8(l - t'), (56) 

and suppressing the ^-dependence in (55), we see that the e„ are in- 
dependent gaussian variables with variances given by 

var(e n ) = N . (57) 

The probability P e may now, in accordance with Section II, be written 

as 

P = _J_ rJa-Ti . ' 

2iri ./-oo a + it x V 1 - zwX„ 

[2 ~] ' ' 

1 ^-> we n ~ 
- m ? ^+w.. • 

* We assume without loss of generality that the ip„(t) are real. 
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or 

1 



exp 



2N f(yo) 



(59) 



V2t 2/»VnU + 2/oX») V-f'M 

In (59) we have written 

( ) = V ye* = V Jh^L. (60) 

and ?/o is determined by 

/« = 0. (61) 

We will pay particular attention to the function f(y), since it is this 
function which determines the exponential behavior for large S/N. We 
note that in terms of the system operator G (60), which defines the 
function f(y), may be written 

™ -(*!&»•) <62) 

where we use the usual notation (a,b) for the inner product of two vec- 
tors in Hilbert space. In (02) we have used s to denote the vector 
s(t — v), t fixed. If one knew explicitly the resolvent operator (1 + yG) 
appearing in (62), one might perhaps calculate the required integrals in 
(62) and search for the maximum of this function of y, thus determining 
?/o which is implicitly a functional of the signal s. In general, however, 
approximation methods must be used. 

Before mentioning some approximation schemes, we would like to 
demonstrate an interesting result. The saddle point y is determined by 
(61 ), or, using (62), we have the implicit relationship 



?/o = 



( S T^? S ) 



(63) 



( S '(l + ? /oG)* S ) 
Thus, from (62) and (63), 

f(yo)=y °( s ' (i + y*Gy s )> (64) 



or 
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where 



Now clearly, 



/(>/o) = 



K = 



(s,Ks)- 
(s,KH) 



G 

l + y G 



fhh) = 



sJCs) 2 . (s,s)(Ks,Ks) 



,s,Kh) 



< 



(Ks,Ks) 



= (s,s) 



(65; 



(66) 



(67) 



by hermiticity of K and Schwarz's inequality. Thus, if the signal energy 
E = (s,s) is fixed, the best performance would be achieved if (l/E)f(yo) 
= 1, provided that this is possible. Certainly the equality in (67) is 
satisfied if s is an eigenfunction of K, and hence of the system function 
G, and at first glance this would appear to be an optimizing solution. 
However, such a solution violates the necessary condition that the 
function f(y) have poles at the nearest eigenvalue for both positive and 
negative y, and hence our basic relation (65) does not hold for such a 
choice of s. In fact, under the assumptions for which (65) was derived, 
no function s, subject to the constant energy constraint, will yield a 
stationary value for the exponent (65).* In fact, it is not difficult to 
convince oneself that the best function s to take (for a positive output) 
is the eigenfunction <p with the largest positive eigenvalue. Note that 
(65) is not applicable here. However, the exponent may be estimated 
according to the discussion of Section III to be 



y = 



1 
X_ 



r- I I = ei 



X _± 
X_ 



1 + 



(68) 



where X + (X_) is the positive (negative) eigenvalue of G with largest 
magnitude. If X+ is large compared with | X_ | , the equality in (67 ) may 
be approached (note e x 2 = (s,s)) for positive pulses. However, for 
negative pulses the factor in the exponent will be 



X_ 



1 + 



x_ 



(69) 



* Even though the exponent is a function of y which implicitly depends on s 
this dependence can be ignored in a first-order variation because of (61). 
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and will become arbitrarily bad. Thus, if one wants symmetry between 
positive and negative pulses, one must take X+ and X_ to have essen- 
tially the same magnitude, leading to a result that is a factor of two 
worse than suggested by the equality in (67 ) . We might point out that 
the value of the exponent that would be obtained if the equality in 
(67) held, is the same as that for the probability of error for an ideal 
correlator, the optimum detector for this binary situation. Since, as 
just described, the optimum binary scheme here uses orthogonal sig- 
nals, the factor of two worse than ideal binary correlation detection is 
not surprising. Since the ideal binary system (again a correlator) for 
orthogonal signals also gives an exponent worse by a factor of two, no 
exponent could be better. 

VT. EXAMPLE OF QUADRATIC DETECTOR 

A particular specialization of the general quadratic detector given 
schematically in Fig. 1 is a differential detector. By this term we shall 
mean that the filters F r and F 2 in Fig. 1 are identical except that F 2 , 
in addition to representing the channel as F x does, has a delay of one bit 
interval. The filter F 3 is a low-pass filter. We treat the simplified base- 
band case where i^i is an ideal low-pass filter with cutoff ft rad/s., and 
F 2 is identical to F t except for a delay T\ , and finally the postdetection 
filter F$ is an ideal integrator with integration time T seconds. The 
analysis is also relevant to the situation where Fi and F 2 are bandpass 
filters, symmetrical with respect to some carrier frequency, provided 
one neglects the double carrier-frequency terms at the input to Fa . 

We begin by considering first the alternative version of the output 
(51), namely 

Eout(t) = h[(s d ,gs) + (s,gs d )], (70) 

where in (70) we have for convenience included the delay Ti directly 
in the signal rather than in the system function, and have denoted the 
delayed version of s by s d . In (70), g denotes the system function for 
two identical filters Fi and F 2 . Equation (49) now reads 

, . [ T '~ sin £2 (u — z) sin ft (v — z) , ,,,, 

q(u,v) = I -, r t r — az. (71) 

* v Lti-?. w(u — z) r(v — z) 



We would like to evaluate the integral for u and v on the entire real line. 
We shall first present a formal evaluation for u and v both restricted to 
the interval (-T/2, T/2), and shall then invoke analytic continuation 
to claim that this restriction on the evaluation of (71) can be dropped. 
Let \f/ n (t) be the prolate spheroidal function normalized to unity in the 
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infinite interval, and hence to X„ on the interval (-T/2, T/2). Also 
let \f/„' (t) be the same function normalized to unity on the interval 
{ — T/2, T/2) and hence to 1/X„ on the infinite interval. The functions 
\j/ n (t) satisfy the equation 14 



L 



772 



8111 



fi (u - Z] 



4, n '( Z ) dz = Xn^n'M, (72) 

J-T/2 t(U — Z) 

where the eigenvalues X„ are real and positive, and ^„(z) satisfy the 
identical equation. On the Hilbert space of square-integrable functions 
on the interval { — T/2, T/2), it then follows that we may write 

sin $2 {u — z) v 1 \ / ' 1 \ 1 ' t \ t~!o\ 

; r- 1 = 2L *»&• KUWn U) ■ (73) 

ir{u — z) »=o 

Therefore, 

g{u,v) = Z Z X»W„'(wW(y) ^ n '{z)^{z) dz, 

h=0 ro=0 J-T/2 

or 

g(u,v) = Ex„V»'(w¥«'(»)- (74) 

«— o 

Reinterpreting (74) to hold on the infinite interval, we have finally 

g{u,v) = Exi B («)^(ti). (75) 

»— o 

Rewriting (70) as 

A' out - \[{8 } U + (T x )§s) + {s,gU{T x )s)l (76) 

where U(Ti) represents the unitary operator of time translation by 
amount 7\ , and comparing (76) with (51), we see that the symme- 
trized system function G{u,v) is given by 

G{u,v) = h Z *■#.(« + T x )Mv) 

" (77) 

Important simplifications in the result (77) obtain when one makes use 
of the fact that the X„ tend to zero rapidly after a few terms. Thus, one 
would expect that the infinite sum for the system function (77) may be 
effectively truncated after a few terms, and the problem of finding the 
resolvent kernel (1 + yO)~ l in (62) is thereby reduced to inverting a 
finite dimensional matrix. The rapidity with which the X„ decrease 
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depends on the parameter 14 c = fiT/2. Some fairly typical situations 
correspond roughly to c = t and 7\ = T. The first six eigenvalues for 
c = t, which are 0.981, 0.749, 0.243, 0.025, 0.001, ~10~ 5 , illustrate this 
behavior well. Furthermore, it usually happens that the shape of the 
signal during the integration time of the filter, i.e., the pulse shape, 
bears a great deal of resemblance to ipo(t), which one may crudely 
visualize as having a (sin t)/t shape. This tends to emphasize the n = 
term in (77 ) even more. Hence we should not be far wrong if we simply 
write (for T x = T) 

G(u,v) = |Xo^o(« + T)Mv) + hUMu)M» + T), (78) 

or equivalently if we define 

• r/2 

€ 



+o(u)Mu + T) du, (79) 

772 



we have that 



GM - — L V2(l + e) J L V2(l+e) J 

(u) - Mu + T) ~\ V Mv) ~ Mv + T) l 
V2(l - 7) _||_ V2(l-6) J' 



The latter form (80) explicitly exhibits the eigenvalues and eigenf unc- 
tions to this approximation. For 7\ = T, e is quite small and in the 
spirit of our approximation may be neglected. We note in closing that 
we have found the representations (78) through (80) extremely useful 
in evaluating the effects of an added external tone on differential phase 
detection of a signal accompanied by gaussian noise. We emphasize that 
the dependence of the degradation on the frequency of the tone in such 
a problem is strongly influenced by the presence of the postdetection 
filter and hence its inclusion (aside from its role of selecting only differ- 
ence frequencies) was essential. This, in fact, motivated much of the 
present work. 

APPENDIX 

Proof of Monotonicity of Exponent on the Contour 
The function of interest is 



exp [-/(w)/tr 2 ] = exp 



N 2 -[ 

u + J 



(81) 
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If we let (a = x + iT with x real and with the saddle point occurring at 
co = iY, we have on the contour L 2 

, yf U + r (r + LY|| 

exp|-/( u )/<, 5 || = exp^-i,E— L , V ,r J ^- (82) 

Therefore, it is sufficient for us to prove for those j such that yf^ 0, that 

.r + rYr + U r(r + L 

which in turn amounts to showing 



1_ 



> 0. (84) 



Now, if we merely recall that T > 0, then the inequality is obviously 
true for dj > 0. If we also recall that if dj < then (-l)M > I\ (84) 
obviously holds for dj < 0. 
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